Displacement oriented interaction in computer-mediated reality

ABSTRACT

In various embodiments, computerized systems and methods for displacement oriented interaction with objects in a computer-mediated environment are provided. In one embodiment, the system detects a wearable device moved with a displacement transversal to a longitudinal axis of the wearable device. If the system determines that the displacement is within a displacement range associated with an actionable item, the system may select the actionable item or activate an operation associated with the actionable item, such as modifying an object in the computer-mediated reality environment.

BACKGROUND

Computer-mediated reality (CMR) refers to one's perception of realitybeing mediated through the use of a computing device, e.g., a wearablecomputer or handheld device. Typically, the computing device mediatesthe user's visual perception of the environment. Various CMRtechnologies, such as virtual reality (VR), augmented reality (AR), andmixed reality (MR), enable users to view and perceive computer-generatedtwo-dimensional (2D) objects and three-dimensional (3D) objects, as ifthe objects were actually present within the user's perceivedenvironment. Advancements in computing have fueled tremendousdevelopment efforts to apply these technologies to both entertainmentand enterprise marketplaces.

Conventional interaction techniques in CMR are often cumbersome, andsuch interactions usually are limited to eye gaze or hand operations,e.g., on a controller. Consequentially, conventional interaction in CMRmay just be another sedentary task or potentially another contributingfactor to risks of metabolic syndrome, heart attack, stroke risk, andoverall health issues, especially for people working long hours withoutsufficient physical activities.

SUMMARY

Embodiments described herein provide methods and systems fordisplacement oriented interaction (DOI) in CMR environments. Morespecifically, the DOI technology disclosed herein can detect a patternof displacement of a wearable device or a user. Subsequently, the DOItechnology may activate a relevant operation in the CMR environmentbased on the pattern of displacement.

In various embodiments, signals indicating a pattern of displacement ofa wearable device or a user are received by a computing deviceconfigured to mediate the CMR environment. The computing device isintegrated with the wearable device in some embodiments. The computingdevice determines whether the pattern of displacement is consistent witha predefined pattern of displacement of an operation related to anobject or a view in the CMR environment. Further, the computing devicecan activate the operation in the CMR environment in response to thedetected pattern of displacement being consistent with the predefinedpattern of displacement of the operation. Optionally, the computingdevice may augment the CMR environment with a presentation of relevantoperations based on the context of the object or the view in the CMRenvironment, e.g., to guide the user to selectively activate a specificoperation. To this end, disclosed methods and systems for DOI enable notonly intuitive and guided interactions with objects in CMR environments,but also enable users in the CMR environments to be physically activeduring such interactions.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used in isolation as an aid in determining the scope of the claimedsubject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in detail below with reference to theattached drawing figures, wherein:

FIG. 1 is a schematic showing an exemplary environment enabled for DOIin CMR, in accordance with some embodiments of the present disclosure;

FIG. 2 is a block diagram of an exemplary computing system configuredfor DOI in CMR, in accordance with some embodiments of the presentdisclosure;

FIG. 3 is a schematic showing a part of an exemplary implementation ofDOI in CMR, in accordance with some embodiments of the presentdisclosure;

FIG. 4 is a schematic showing another part of the exemplaryimplementation of DOI in CMR, in accordance with some embodiments of thepresent disclosure;

FIG. 5 is a schematic showing another exemplary implementation of DOI inCMR, in accordance with some embodiments of the present disclosure;

FIG. 6 is a flow diagram showing a process for DOI in CMR, in accordancewith some embodiments of the present disclosure;

FIG. 7 is a flow diagram showing another process for DOI in CMR, inaccordance with some embodiments of the present disclosure; and

FIG. 8 is a block diagram of an exemplary computing environment suitablefor use in implementing embodiments of the present disclosure.

DETAILED DESCRIPTION

In accordance with embodiments described herein, the CMR environment caninclude any one of a virtual reality (VR) environment, an augmentedreality (AR) environment, and a mixed reality (MR) environment, each ofwhich are at least partially rendered by one or more computing devicesand provided for immersive display to a user. The term “environment” canreference a portion of or an entire collection of rendered objectsavailable for perceivable display to the user. For instance, in a fullyvirtual environment (e.g., VR), the environment can reference one ormore sets of rendered objects, a rendered background or particle system,or any portion and/or combination thereof. In a partially virtualenvironment (e.g., AR or MR), the environment can reference one or moresets of rendered objects, a particle system, a real-world space orbackdrop, or any portion and/or combination thereof.

In dynamic and constantly changing CMR environments, it is important toprovide users with intuitive modes of interaction with precision incontrollability. Conventional interactions, such as using keyboards andmice, have become prevalent in conventional computing environments.However, such conventional interaction means are disadvantageous andoften impractical for a user to achieve an immersed experience in a CMRenvironment.

Embodiments of the present disclosure provide systems and methods forDOI in a CMR environment. With the DOI technology, a user can interactnaturally with a target object in the CMR environment using kineticmovement. In some embodiments, a wearable device, such as a head-mounteddisplay (HMD) device, can detect the kinetic movement and recognize thepattern of displacement of the wearable device. If the pattern ofdisplacement is consistent with a predefined pattern of displacement ofan operation related to an object or a view in the CMR environment, thewearable device may further activate the operation in the CMRenvironment. Optionally, the computing device may augment the CMRenvironment with a presentation of relevant operations based on thecontext of the object or the view in the CMR environment, e.g., to guidethe user to selectively activate a specific operation.

By way of example, the wearable device detects the eye gaze of a user ata 3D object. Accordingly, the wearable device may cause a radial menuwith multiple sections presented within the field of view of the user inthe CMR environment. Each section of the radial menu represents a modeor an operation related to the 3D object. Further, the wearable devicemay cause a replica of the radial menu (e.g., a light-based replica)displayed on the floor around the user.

While guided by the radial menu in front of the user and remaining atthe head-up posture, the user may walk to a specific section of theradial menu to select a specific mode or operation related to the targetobject. In this regard, the wearable device may detect its owndisplacement caused by the kinetic movement of the user, and determinethe specific section of the radial menu selected by the useraccordingly. If the wearable device confirms the user's section of thespecific mode or operation, the wearable device may further activate themode or operation, such as to modify the target 3D object in the CMRenvironment.

Advantageous, disclosed methods and systems for DOI in CMR enable notonly intuitive and guided interactions with objects in CMR environments,but also enable users in the CMR environments to be physically activeduring such interactions. Further, the DOI technology may also free upuser's hands, so that users may use hand operations (e.g., handgestures) in combination of foot movement to engage more sophisticatedinteractions in CMR environments.

With reference now to FIG. 1, an exemplary environment enabled for DOIwith objects in a CMR environment is provided, in accordance with someembodiments of the present disclosure. In this CMR environment,head-mounted display (HMD) 120 enables user 110 to view virtual object150. HMD 120 can detect the gaze of user 110, and select virtual object150 based on the gaze of the user 110.

User 110 may want to interact with virtual object 150, e.g., for thepurpose of modifying virtual object 150. In this case, user 110 may makea preconfigured kinetic gesture, such as a jumping gesture, a squattinggesture, or a hand gesture, to activate a menu of available operationsassociated with virtual object 150. The kinetic gesture can be detectedby HMD 120 and/or a computing device (e.g., stationary sensors in theroom, another wearable device, etc.) connected or in communication withHMD 120.

Subsequently, HMD 120 may cause radial menu 130 with different sectionspresented within the field of view of the user in the CMR environment.Each section of the radial menu represents an operation associated withvirtual object 150. The content of radial menu 130 is context-dependent,such that operations relevant to the present view and/or the presentobject are presented on radial menu 130. For example, operations relatedto virtual object 150 may include changing colors, changing orientation,changing size (e.g., zoom in or zoom out), changing shape, moving,rotating, adding data, removing data, etc.

Further, HMD 120 may replicate radial menu 130 as replica menu 140, andcause replica menu 140 displayed on the floor around the user. Replicamenu 140 may be displayed on a large display placed horizontally on thefloor, as an actual image projected from a light emitter, or a virtualimage rendered by HMD 120. Radial menu 130 or replica menu 140 canserver as an intuitive visual guide for user 110 to select one or moreoperations associated with virtual object 150, such as user 110 may keepup with a head-up posture for interacting with the CMR environment.

By way of example, user 110 may walk to section A or B on replica menu140 to select a specific operation associated with section A or B. User110 may select both operations associated with section A and B bywalking to these two sections sequentially or standing on both sectionswith both feet simultaneously. User 100 may also choose to exit thepresent mode of menu selection, e.g., by making another preconfiguredkinetic gesture, such as jumping or squatting, to exit from radial menu130 and/or replica menu 140.

If a menu item is selected, HMD 120 may cause the selected menu item tobe rendered differently. The selected menu item may be highlighted orchanged to a different presentation, so that user 110 may receive thevisual feedback or confirmation of the menu selection. In otherembodiments, feedback could also include audible feedback, hapticfeedback, or feedback in other forms.

If a menu item is selected, HMD 120 may mediate the CMR environment intoa different mode corresponding to the selected menu item. Sometimes, thedefault behavior associated with the selected menu item is activated.Sometimes, additional input may be required to active the operationassociated with the selected menu item. The default behavior or requiredadditional input may depend on the context and the characteristics ofthe target object. Each menu item may be viewed as an actionable item inthe CMR environment. Each menu item may represent one or more operationsapplicable in the CMR environment.

By way of example, menu item A may represent a mode of zooming, whilemenu item B may represent a mode of panning. Accordingly, if the mode ofzooming is activated, a default zooming operation may be automaticallyapplied to virtual object 150. Alternatively, user 110 may use furthercontrol mechanisms, such as hand gestures, to provide a control factorfor the operation of zooming with virtual object 150.

In some embodiments, radial menu 130 and/or replica menu 140 areoptional if user 110 does not need any visual guide, such as for routineoperations. As an example, user 110 may configure HMD 120 to zoom in atarget object by walking toward the target object, conversely, to zoomout the target object by walking away from the target object.

With reference to FIG. 2, an exemplary device 200 configured for DOI inCMR is provided, in accordance with some embodiments of the presentdisclosure. In some embodiments, device 200 is embodied as a HMD device,like HMD 120 illustrated in FIG. 1. In some embodiments, device 200 mayinclude any type of computing device, such as computing device 800described below with reference to FIG. 8. As will be described in moredetail below, in various embodiments, device 200 can include, amongother things, kinetic detector 210, context handler 220, patternrecognizer 230, action operator 240, and rendering component 250.

In further embodiments, kinetic detector 210 can include, among otherthings, displacement sensor 212 and orientation sensor 214. Inaccordance with embodiments described herein, it is contemplated thatthe aforementioned different components in device 200 can be implementedin any one or more components or subcomponents of device 200. Forinstance, any one of kinetic detector 210, context handler 220, patternrecognizer 230, action operator 240, and rendering component 250 may beimplemented at least in part within a processor, graphical processingunit (GPU), application code, firmware, and the like.

Device 200 can enable interaction with a holographic world by detectingthe displacement of device 200 itself, another wearable device, or theuser of device 200. The pattern and measurement of the displacement maybe translated into one or more operations in CMR. When device 200 isembodied as a HMD device, a user can interact on data or hologramvisualization with their head movement while staying in context.Further, a radial menu may be displayed on the floor or in front of theuser, and the user can interact with the CMR by walking while keeping upa head-up posture. The menu may be displayed in other shapes, forms, orconfigurations. Options of the menu items may be selected based on thecontext, such as the object gazed by the user. In one embodiment, device200 determines the selected object for the operation just before theuser moves to a menu selection mode.

In one embodiment, head movement is mapped to interactions in the CMRenvironment as follows. First the HMD is registered with a referencelocation. Second, the orientation and the displacement of the HMD areconverted into an amount of modification to an object in the CMRenvironment. This displacement can be an absolute displacement ascomputed from the difference between the reference point and the currentposition of the HMD. This displacement can also be computed as relativedisplacement, such as a variation in time. As an example, the relativedisplacement will increase by one unit per second.

In various embodiments, the HMD defines a longitudinal axis that isdiagonal to the plane of the body of the device. In general, thislongitudinal axis is in parallel with the longitudinal axis of the headof the user when the user keeps the head-up posture. The HMD thendetects displacement transversal to the longitudinal axis when the HMDmoves from one position to another position. In one embodiment, the menuis displayed based on the user's orientation.

Device 200 is configured to capture and measure the displacement ofitself or its user, e.g., based on displacement sensor 212 andorientation sensor 214. Displacement sensor 212 may detect an absolutionlocation, e.g., GPS based location, or a relative location, e.g., anindoor location based on one or more reference points. Displacementsensor 212 may include a GPS sensor, a WiFi receiver, an infraredsensor, etc. to detect absolute location or relative location.Displacement sensor 212 may use other external sensor within theenvironment such as Kinect® sensors fixed in the room or the like.

Device 200 may use the direction and distance between two locations todetect the displacement of device 200. Displacement sensor 212 mayfurther include an accelerometer or a pressure sensor to detect jumpingor squatting motion, and/or stepping, which appears as a displacementparallel to the longitudinal axis. In response to the displacementparallel to the longitudinal axis, device 200 may cause a menu to bedisplayed or removed from the CMR environment.

In various embodiments, threshold values for displacement may beconfigured by a user or system to avoid triggering unwantedinteractions. The threshold values for displacement may also bedynamically learned as the user is using the device, and/or through acalibration procedure. In this way, only significant displacement ofdevice 200 is recognized as intentional user interaction by kineticdetector 210.

Orientation sensor 214 may detect the orientation of device 200. Invarious embodiments, orientation sensor 214 may include a gyroscopesensor or a magnetometer to determine the orientation of device 200. Byway of example, based on the central gravitational pull, orientationsensor 214 may detect whether device 200 is up or down. Orientationsensor 214 may include a variety of technologies, such as optical ormagnetic means to measure rotation. In some embodiments, the initialorientation of device 200 is registered as the starting point.Orientation sensor 214 can then tell how device 200 may have rotatedfrom the starting point. A change of the orientation of device 200 mayindicate its movement. Further, the orientation of device 200 mayfurther indicate the focus of the user. For example, if there are twomenus in the CMR environment, the orientation of device 200 may help todiscern which menu is focused by the user.

In some embodiments, orientation sensor 214 determines an angulardirection of the displacement in reference to the first position beingan origin of a two-dimensional Euclidean plane transversal to thelongitudinal axis. In reference to FIG. 1, the original position may bedefined based on the original standing position of user 110 on thefloor. Further, an X-axis may be defined based on the axis connectingthe two eye displays of HMD 120. Similarly, the Y-axis may be definedbased on the axis orthogonal to the X-axis on the same plane.Accordingly, the Z-axis may be defined as the axis orthogonal to boththe X-axis and the Y-axis in the 3D space. In this way, the angulardirection of the displacement of HMD 120 can be measured in reference tothe X-axis or the Y-axis.

Kinetic detector 210 may include various other sensor component(s),e.g., a gyroscope, an accelerometer, and a magnetometer, infraredlights, infrared cameras, motion sensors, light sensors, 3-D scanners,CMOS sensors, a GPS radio, etc. In various embodiments, kinetic detector210 can employ the aforementioned sensors, among other things, toidentify an intended target (e.g., a hand or a foot) of the user ofdevice 200, and track physical locations and movements of device 200 oranother device in communication with device 200.

Accordingly, from the received signals, kinetic detector 210 maydetermine various characteristics associated with the movement of device200 or various body parts (e.g., the head, a hand, or a foot) of theuser of device 200. The characteristics may include the velocity, theacceleration, the direction, the distance, the path, the duration, thefrequency, etc. of the displacement of device 200. In variousembodiments, such characteristics may form a pattern recognized bypattern recognizer 230.

In some embodiments, kinetic detector 210 can recognize a gestureperformed by the user. For instance, the user can perform gestures usinghis extremities (e.g., arms, fingers, legs, head, etc.), which can bedetected, by way of example, via an optical input component such as acamera, motion sensor, infrared sensor, and the like. In variousembodiments, such gestures can be converted into control parameters ofan action directed by action operator 240.

Device 200 may include a depth camera and/or other sensors (not shown)that support understanding elements of a scene or environment, forexample, generating a 3D mesh representation of a surrounding real-worldenvironment. Further, kinetic detector 210 can use the depth cameraand/or other sensors to detect interactions, e.g., between a foot and aradial menu on the floor.

Context handler 220 gathers and analyzes contextual information relatedto the CMR environment in general or the specific object(s) in the CMRenvironment. Based in part on the analyzed contextual information,pattern recognizer 230 can more accurately identify the displacementpattern, and action operator 240 can more accurately identify the actionto be executed.

Context handler 220 can analyze various contextual information,including stored data, processed data, and/or obtained input data. Inputdata can include sensor data, controller data, input device data, or anyother obtained electronic data that related to one or more virtualobjects rendered in the CMR environment. For example, electronic dataindicating a user selection from one or more rendered objects can beprocessed to identify the intended interaction target.

By way of example, one type of contextual information includes thecharacteristics of the CMR environment, such as the type of interactionsand actions available in the CMR environment. Another type of contextualinformation includes the characteristics of the intended object forinteraction, such as the options of interactions and actions availableto the object. Another type of contextual information includes therelationship between the present location of device 200 and the intendedobject for interaction, or the user's relative proximity to the intendedtarget. In this way, a comparison of the relative distance can be madeagainst a threshold distance.

In various embodiments, contextual information can be derived from dataor electronic information associated with the intended target, theimmersive environment, other rendered objects, real world objects,sensor data, a user profile, an application state, or any combinationthereof, to enable pattern recognizer 230 to determine the displacementpattern, and to enable action operator 240 to determine the action toexecute. The foregoing examples are merely exemplary and are notintended to be limiting in any way.

Pattern recognizer 230 can use the contextual information todisambiguate the user intention and more accurately interpret thedisplacement pattern. Advantageously, when there are many options tointerpret the user intent associated with a kinetic gesture, patternrecognizer 230 may utilize the contextual information to limit itsconsideration to only relevant displacement patterns.

In various embodiments, pattern recognizer 230 can determine pertinentdisplacement patterns of the HMD based on contextual information relatedto the target object or the view in the CMR environment. As an example,the contextual information provided by context handler 220 may revealthat the user is browsing a list of merchants in the entrance of avirtual mall by gazing at the virtual directory. In this case, patternrecognizer 230 may limit the scope of relevant displacement patternsrelated to the virtual directory, rather than other options related tothe parking lot of the virtual mall.

Pattern recognizer 230 may utilize the contextual information tointerpret or disambiguate the kinetic gesture. As an example, the usermay start to walk forward, and kinetic detector 210 can detect thishorizontal displacement of device 200. However, if the context handler220 identifies a scene change in the CMR environment which normallyrequires the user to walk to another position, then pattern recognizer230 may interpret the displacement as an incidental event, rather thanan intentional kinetic gesture.

In various embodiments, pattern recognizer 230 may identify thedisplacement pattern based on the data from kinetic detector 210 andcontext handler 220. By way of example, kinetic detector 210 detects avertical displacement, and context handler 220 identifies an objectgazed by the user of device 200 and the contextual information ofavailable actions associated with the object. In this case, patternrecognizer 230 may interpret the vertical displacement as the userintent to activate or disable the menu associated with the object via akinetic gesture (e.g., jumping or squatting).

As another example, kinetic detector 210 detects a horizontaldisplacement in a particular direction, and context handler 220identifies a radial menu displayed in the CMR environment. In this case,pattern recognizer 230 may interpret the horizontal displacement as theuser intent to select a particular menu item on the radial menu.

As yet another example, kinetic detector 210 detects a hand gesture ofthe user, and context handler 220 identifies a particular radial menualready being selected. In this case, pattern recognizer 230 mayinterpret the hand gesture as the user input of control parameters forthe action associated with the selected menu item, and pass the controlparameters to action operator 240.

Action operator 240 can process information from pattern recognizer 230,and can further determine how the CMR environment should be modified.Action operator 240 may determine a form of the modification based onthe displacement pattern identified by pattern recognizer 230. It mayalso be said that action operator 240 can determine the mode for thevirtual object to be modified or the reaction of the virtual object inresponse of the displacement of device 200. Further, action operator 240may also determine the control parameters for the modification.

Continuing with the examples previously discussed in connection withpattern recognizer 230, when pattern recognizer 230 interprets thehorizontal displacement as the user intent to select a particular menuitem on the radial menu, e.g., zooming in, action operator 240 mayfurther determine the zooming factor based on the measurement of thedisplacement, such as making the zooming factor proportional to thedistance of the displacement. By the same token, if the selected menuitem corresponds to a rotation action, action operator 240 may furtherdetermine the angle of rotation based on the measurement of thedisplacement, such as making the angle of rotation proportional to thedistance of the displacement.

In some embodiments, action operator 240 is further configured toretrieve the menu, e.g., based on the contextual information provided bycontext handler 220. As an example, if the user is gazed on an object,then the menu related to the object may be retrieved by action operator240.

In some embodiments, action operator 240 is further configured to causethe menu being removed from a present view of the CMR environment inresponse to the displacement being outside of all displacement rangesassociated with the menu. For example, the menu may be activated anddisplay to the user related to the object gazed by the user. However,the user may choose to keep on walking instead of making a recognizablekinetic gesture. In this case, the distance of the displacement is goingto go beyond all displacement ranges associated with any menu item.Accordingly, action operator 240 may work in synergy with patternrecognizer 230 to remove the menu from being displayed when the userappears uninterested in the menu.

Rendering component 250 may render virtual objects or images based atleast in part on the 3D mesh representation. In this regard, device 200can specifically include functionality (e.g., augmented reality or mixedreality experiences realized through rendering component 250) to modifythe CMR environment (e.g., to create or modify a virtual object) basedon the action identified by action operator 240.

In various embodiments, rendering component 250 may cause options ofinteraction to be presented in the CMR environment, such as a radialmenu displayed in front of the user or on the floor, so that the usermay be presented with a visual guide listing the available interactionoptions associated with the target object.

If the menu is to be displayed as a radial menu, then each menu item isgiven a direction. When the rendering component 250 causes the menu topresent in a radial fashion in the CMR environment, the menu items wouldbe arranged in the radial fashion according to respective directionsassociated with the menu items.

Further, rendering component 250 can modify how virtualized objects arerendered and ultimately perceived by the user, e.g., based at least inpart on the action identified by action operator 240. In other words,rendering component 250 may modify the virtual object, e.g., bymodifying the configuration (e.g., the position, size, shape, angle) ofthe virtual object, transforming (e.g., scale, rotate, skew, stretch,warp) the virtual object, or deleting the virtual object, or evencreating new ones, in accordance with the action identified by actionoperator 240.

In various embodiments, each virtual object rendered by renderingcomponent 250 is positioned at a corresponding location relative todevice 200, and thereby also to the user wearing device 200. Moreover,the rendered virtual object can be continuously modified in real time,by rendering component 250, to maintain the proper perspective andconfiguration of the rendered virtual object.

In various embodiments, action operator 240 or rendering component 250generates an indication for feedback. The indication for feedback may berendered by rendering component 250 as a visual indicator, such as acolor change of the selected menu or a visual presentation or preview ofthe action. The indication for feedback may be presented by actionoperator 240, such as haptic feedback, thermal feedback, or audiblefeedback associated with the action. The feedback may be provided viadevice 200 or another device connected to device 200. In this way, theuser may receive confirmation of the interaction with the CMRenvironment.

Device 200 may be a scene-aware device that understands elementssurrounding a real-world environment and generates virtual objects asaugmented reality images. Device 200 may include any type of augmentedreality device. An augmented reality device can be an HMD device, andother types of augmented reality devices (e.g., projectors) arecontemplated in accordance with embodiments of the present disclosure.One non-limiting example of device 200 is the HoloLens®, developed byMicrosoft Corporation of Redmond, Wash.

Further, a component of device 200 as used herein refers to any device,process, service, or any combination thereof. A component of device 200may be implemented using hardware, software, firmware, a special-purposedevice, or any combination thereof. Different components of device 200may be integrated into a single device or it may be distributed overmultiple devices. The various aspects of a component may be collocatedor distributed. The component may be formed from other components andaspects thereof.

Other arrangements and elements (e.g., machines, interfaces, functions,orders, and groupings of functions, etc.) can be used in addition to orinstead of those shown, and some elements may be omitted altogether.Further, many of the elements described herein are functional entitiesthat may be implemented as discrete or distributed components or inconjunction with other components, and in any suitable combination andlocation. Various functions described herein as being performed by oneor more entities may be carried out by hardware, firmware, and/orsoftware. For instance, various functions may be carried out by aprocessor executing instructions stored in memory.

FIG. 3 is a schematic showing a part of an exemplary implementation ofDOI, and FIG. 4 is a schematic showing another part of the exemplaryimplementation of DOI, in accordance with some embodiments of thepresent disclosure. Menu 330 is a radial menu rendered by HMD 320 infront of user 310. Menu 330 has 8 different sections with respectivedirections. Menu 340 mimics menu 330 and is displayed on the floor withthe same menu items in 8 directions, i.e., forward, backward, side left,side right, and 4 diagonal directions.

From blocks 360 and 370 in FIG. 3, HMD 320 confirms user 310's selectionof a particular menu item. At block 360, user 310 steps back with onefoot to section E. HMD 320 detects its own displacement patterncorresponding to the kinetic gesture made by user 310 or directlydetects the foot movement of user 310, e.g., based on optical sensors ora connected wearable device on user 310's foot. In some embodiments,sensing of the foot movement could be done based on sensor data from HMD320, cameras mounted on HMD 320, external depth cameras, etc. Inresponse to the detected displacement pattern, HMD 320 provides visualfeedback to user 310, e.g., by causing visual effect 332 to section E onmenu 330 to confirm that user 310 is attempting to select section E.

At block 370, user 310 moves another foot to section E, effectivelychanging the location of HMD 320 from the original reference point toanother location within section 342 of menu 340. HMD 320 detects thisdisplacement pattern. In response to the detected displacement pattern,HMD 320 provides visual feedback to user 310, e.g., by causing anothervisual effect 334 to section E on menu 330, such as putting a stararound the content on section E, to confirm that section E has beenselected.

At blocks 380 and 390, user 310 moves back to the original startingposition. In this case, HMD 320 detects this displacement pattern asmoving from the first position (i.e., the original position) to thesecond position (i.e., the position on section E), followed by anothermovement from the second position back to the first position. HMD 320may interpret this displacement pattern as a gesture to activate theoperation associated with the selected menu item. As an example, ifsection E represents the operation of rotation, after user 310 returnsback to the original position, the object gazed by user 310 may start torotate.

In this sequence of foot movement, HMD 320 detects the pattern of footmovement having two consecutive moves with a first move from a firstposition to a second position, and a second move from the secondposition to the first position. Accordingly, HMD 320 selects a menu itemof a menu based on the direction formed from the first position to thesecond position, which is the direction pointing to section 342 in thiscase.

In an alternative embodiment, from the sequence of blocks 360, 380, and390 in FIG. 3 and FIG. 4, HMD 320 may confirm user's intent to withdrawa potential selection of menu 330. As an example, at block 380, insteadof moving his right foot to section E as at 370, user 310 moves his bodyweight to his right foot, and starts to move back his left foot fromsection E to the original reference point. HMD 320 detects thisdisplacement pattern and causes another visual effect 336 to section Eon menu 330, such as underlining the content on section E, toacknowledge the user's intent to withdraw the selection of section E.

At block 390, user 310 returns to the original reference point. HMD 320detects this displacement pattern and causes another visual effect 338to section E on menu 330, such as removing any special visual effect tosection E, to confirm the user's intent to withdraw any selection onmenu 330.

Accordingly, the displacement patterns associated with blocks 360, 380,and 390 provide a new way to withdraw a potential selection of menu 330without triggering a mode switch or modification to the targeted objectsin the CMR environment. In other embodiments, different displacementpatterns may be registered to HMD 320 as options for user 310 to avoidunintended menu selection, such as by doing two steps in oppositedirections.

FIG. 5 is a schematic showing another exemplary implementation of DOI.In this embodiment, menu 530 is a radial menu rendered by HMD 520 infront of user 510. Menu 530 has 8 different sections with respectivedirections. Menu 540 mimics menu 530 and is displayed on the floor withthe same menu items also in 8 directions, i.e., forward, backward, sideleft, side right, and four diagonal directions.

HMD 520 utilizes hand gesture 512 in addition to the displacementpattern of HMD 520 to cause object 552 to be changed to object 554. User510 may use hand gestures to operate on the selected menu item from menu530, e.g., zooming on data if a zoom mode is selected. In this case,user 510 steps forward to section 542 of menu 540. HMD 520 detects thisdisplacement pattern caused by this foot movement. Further, HMD 520 alsodetects hand gesture 512, which is a vertical downward swiping motion,based on optical sensors or a connected wearable device on user 510'swrist.

In response to the detected displacement pattern, HMD 520 providesvisual feedback to user 510, e.g., by causing visual effect 532 tosection A on menu 530, indicating that user 510 is attempting to selectsection A, which is the menu item for zooming. In further response tothe detected hand gesture 512, HMD 520 may execute an action, which isto shrink object 552, and render it as object 554. The characteristicsof hand gesture 512 may be used by HMD 520 to control the transformationfrom object 552 to object 554, such as using the distance or speed ofthe vertical downward swiping motion as control parameters for shrinkingobject 552.

In alternative embodiments, instead of making hand gesture 512, user 510may use the characteristics of his foot movement to control themodification to object 552. By way of example, user 510 may use the steplength to control the shrinking factor for object 552, in whichdifferent step lengths may correspond to different shrinking factors.

Referring now to FIG. 6 in light of FIGS. 1-5, FIG. 6 is a flow diagramshowing a method 600 for DOI in a CMR environment. Each block of method600, and other methods described herein, comprises a computing processthat may be performed using any combination of hardware, firmware,and/or software. For instance, various functions may be carried out by aprocessor executing instructions stored in memory. The methods may alsobe embodied as computer-usable instructions stored on computer storagemedia. The methods may be provided by a stand-alone application, aservice or hosted service (stand-alone or in combination with anotherhosted service), or a plug-in to another product, to name a few.

At block 610, a displacement of a wearable device may be detected, bykinetic detector 210 of FIG. 2. If the distance of the displacement isbelow a predetermined threshold, the duration of the displacement isbelow a predetermined threshold, or another characteristic of thedisplacement is not meeting a requisite, the detected displacement maynot warrant further processing.

In some embodiments, a hysteresis process is used to detect displacementor suspend an ongoing DOI. For example, the user may be required to movethe wearable device (e.g., a HMD) in any direction but with sufficientspeed (e.g., 20 cm per second) to trigger a detection of displacement.Alternatively, the user may be required to move the wearable device pasta certain threshold distance from the idle position (e.g., 20 cm). Thethreshold distance may be set differently for each foot. The thresholddistance may be set differently for different foot movement, such asmoving toward a menu item or moving back to the center.

As another example, an ongoing DOI may be suspended when the user stopsto move for a period (e.g., 1 second). As yet another example, a controldevice may be used to activate or stop DOI, such as using a button on acontrol device to activate or stop DOI.

At block 620, the displacement may be compared with a pattern ofdisplacement associated with an operation, e.g., by pattern recognizer230 of FIG. 2. The specific operation may be defined by a menu item,such as an operation to modify the target object via panning, zooming,rotating, etc. The pattern of displacement may be also defined by themenu item, for example, the placement of the specific menu item on themenu, i.e., in a specific section with a particular direction.

In some embodiments, a menu item represents an operation related to anobject or a view in a CMR environment. Further, the menu item isassociated with a displacement range, which may be a range of angles ina plane transversal to the longitudinal axis defined by the HMD. In someembodiments, this plane transversal to the longitudinal axis is inparallel with the floor.

Using menu 140 in FIG. 1 as an example, section C may be associated withthe range of 0 to 45 degrees; section B may be associated with the rangeof 46 to 90 degrees; section A may be associated with the range of 91 to135 degrees; section H may be associated with the range of 136 to 180degrees; and so forth.

Further, all sections may be associated with a minimum distance and amaximum distance. In one embodiment, the minimum distance and themaximum distance may be dynamically change based on speed oracceleration of the detected displacement. In various embodiments,pattern recognizer 230 of FIG. 2 may be further configured to determinewhether the distance value of the displacement is greater than a minimumdistance and lesser than a maximum distance associated with thedisplacement range. In some embodiments, a displacement outside of thedistance range will be overlooked by the system. In this way, detectionsof false positive displacement may be mitigated. By way of example, ifthe user keeps on walking, the distance of the displacement would exceedthe maximum distance threshold, and this action of walking would not beinterpreted as a kinetic gesture for the purpose of this disclosure.

Pattern recognizer 230 may compare various characteristics of thedetected displacement with the known pattern of displacement associatedwith the operation, such as the direction or the angle of thedisplacement, the duration of the displacement, the speed of thedisplacement, etc. By way of example, one characteristic to be comparedis the direction. If the direction falls into the range of 0 and 45degrees, then section C of menu 140 may be selected.

Different matching criteria may be used, such as identical match orsubstantial match. In an identical matching test, all characteristics tocompare with must match. In a substantial matching test, a predefinedthreshold, e.g., 70% of the compared characteristics, may be used todetermine whether the substantial match test is passed. By way ofexample, if there are three characteristics to compare with, and thethreshold is ⅔; then a menu item would be selected only if two or threecharacteristics are matched between the detected displacement patternand the known pattern of displacement associated with the menu item.

At block 630, the operation can be activated in the CMR environmentbased on the comparison, e.g., by action operator 240 of FIG. 2. Ingeneral, if the detected displacement pattern matches a pattern ofdisplacement associated with an operation, then the operation may beactivated. Further, action operator 240 may use the characteristics ofthe displacement to control the action. By way of example, actionoperator 240 may use the distance of displacement to control themagnitude of the action, e.g., the zooming factor.

In various embodiments, the pattern recognizer is configured todetermine whether the angular direction of the displacement is bound bythe minimum angle and the maximum angle of the displacement range. Asdescribed in connection with block 620, a displacement range associatedwith a menu item may define one or more extrema, such as 0 as theminimum and 45 as the maximum for the range associated with section C ofmenu 140. By way of example, if the detected displacement pattern showsan angle of 23 degrees, then it is bounded by the two extrema of thedisplacement range associated with section C of menu 140. Therefore, thesection C of menu 140 may be selected, or the operation associated withthe section C may be activated.

In some embodiments, a hysteresis process is used to activate theoperation. By way of example, in response to the menu item beingselected, the HMD may modify a visual presentation of the menu item onthe menu, and start a timer with a predetermined time period. After thetimer runs down, the HMD may active the operation associated with themenu item.

In connection with FIG. 3, at block 370, a timer may be started. If thepredetermined time period is over, then the operation associated withsection E may be activated. However, if the user makes another movementbefore the timer runs out, such as walking forward two steps intosection A, the menu item of section A may be selected instead.Subsequently, another timer may be started for this new selection.Advantageously, the user may easily change his menu selection in theseembodiments.

Referring now to FIG. 7 in light of FIGS. 1-6, FIG. 7 is a flow diagramshowing a method 700 for DOI in CMR. Each block of method 700, and othermethods described herein, comprises a computing process that may beperformed using any combination of hardware, firmware, and/or software.For instance, various functions may be carried out by a processorexecuting instructions stored in memory. The methods may also beembodied as computer-usable instructions stored on computer storagemedia. The methods may be provided by a stand-alone application, aservice or hosted service (stand-alone or in combination with anotherhosted service), or a plug-in to another product, to name a few.

At block 710, a signal indicating a foot movement may be received, e.g.,by the kinetic detector in HMD 520. By way of example, in connectionwith FIG. 5, user 510 moves his right foot forward into section 542, anda signal, e.g., generated from the displacement sensor, indicating thisfoot movement is received by the kinetic detector in HMD 520.

At block 720, the pattern of foot movement may be detected, e.g., by thepattern recognizer in HMD 520. Continuing with the aforementionedexample, the pattern recognizer in HMD 520 may recognize the presentpattern of foot movement as a lateral displacement with a forwarddirection, which matches the displacement pattern associated with themenu item on section 542.

At block 730, a menu item in a CMR environment may be selected based onthe pattern of foot movement, e.g., by the action operator in HMD 520.Continuing with the aforementioned example, the action operator in HMD520 may then select the menu item on section A and provide a visualfeedback by changing the appearance of section A.

At block 740, an operation may be activated based on the selected menuitem and another input. In connection with FIG. 5, the additional inputis a hand gesture. The section A may represent a zooming operation, andthe hand gesture may further provide the control factor for zooming,e.g., proportional to the distance or speed of the hand gesture. Inanother embodiment, the additional input may be generated from a controldevice, such as a touchpad that enables the additional input. In otherembodiments, other type of input may be used to provide the controlparameters for the operation.

Having described embodiments of the present invention, an exemplaryoperating environment in which embodiments of the present invention maybe implemented is described below in order to provide a general contextfor various aspects of the present invention. Referring initially toFIG. 8 in particular, an exemplary operating environment forimplementing embodiments of the present invention is shown anddesignated generally as computing device 800.

Computing device 800 is but one example of a suitable computingenvironment and is not intended to suggest any limitation as to thescope of use of the technology described herein. Neither shouldcomputing device 800 be interpreted as having any dependency orrequirement relating to any one or combination of componentsillustrated.

The technology described herein may be described in the general contextof computer code or machine-usable instructions, includingcomputer-executable instructions such as program components, beingexecuted by a computer or other machine. Generally, program components,including routines, programs, objects, components, data structures, andthe like, refer to code that performs particular tasks or implementsparticular abstract data types. The technology described herein may bepracticed in a variety of system configurations, including handhelddevices, consumer electronics, general-purpose computers, specialtycomputing devices, etc. Aspects of the technology described herein mayalso be practiced in distributed computing environments where tasks areperformed by remote-processing devices that are connected through acommunications network.

With continued reference to FIG. 8, computing device 800 includes a bus810 that directly or indirectly couples the following devices: memory820, one or more processors 830, one or more presentation components840, input/output (I/O) ports 850, I/O components 860, and anillustrative power supply 870. Bus 810 represents what may be one ormore busses (such as an address bus, data bus, or a combinationthereof). Although the various blocks of FIG. 8 are shown with lines forthe sake of clarity, in reality, delineating various components is notso clear, and metaphorically, the lines would more accurately be greyand fuzzy. For example, one may consider a presentation component suchas a display device to be an I/O component. Also, processors havememory. The inventors hereof recognize that such is the nature of theart and reiterate that the diagram of FIG. 8 is merely illustrative ofan exemplary computing device that can be used in connection with one ormore aspects of the technology described herein. Distinction is not madebetween such categories as “workstation,” “server,” “laptop,” “handhelddevice,” etc., as all are contemplated within the scope of FIG. 8 andrefer to “computer” or “computing device.”

Computing device 800 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 800 and includes both volatile andnonvolatile, removable and non-removable media. By way of example, andnot limitation, computer-readable media may comprise computer storagemedia and communication media. Computer storage media includes bothvolatile and nonvolatile, removable and non-removable media implementedin any method or technology for storage of information such ascomputer-readable instructions, data structures, program modules, orother data.

Computer storage media includes RAM, ROM, EEPROM, flash memory or othermemory technology, CD-ROM, digital versatile disks (DVD) or otheroptical disk storage, magnetic cassettes, magnetic tape, magnetic diskstorage, or other magnetic storage devices. Computer storage media doesnot comprise a propagated data signal.

Communication media typically embodies computer-readable instructions,data structures, program modules, or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared, and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia.

Memory 820 includes computer storage media in the form of volatileand/or nonvolatile memory. The memory 820 may be removable,non-removable, or a combination thereof. Exemplary memory includessolid-state memory, hard drives, optical-disc drives, etc. Computingdevice 800 includes one or more processors 830 that read data fromvarious entities such as bus 810, memory 820, or I/O components 860.Presentation components 840 present data indications to a user or otherdevice. Exemplary presentation components 840 include a display device,speaker, printing component, vibrating component, etc. I/O ports 850allow computing device 800 to be logically coupled to other devices,including I/O components 860, some of which may be built in.

In various embodiments, memory 820 includes, in particular, temporal andpersistent copies of displacement oriented interaction (DOI) logic 822.DOI logic 822 includes instructions that, when executed by one or moreprocessors 830, result in computing device 800 performing variousfunctions, such as, but not limited to, method 600 or 700. In variousembodiments, DOI logic 822 includes instructions that, when executed byprocessor(s) 830, result in computing device 800 performing variousfunctions associated with, but not limited to, kinetic detector 210,context handler 220, pattern recognizer 230, action operator 240, orrendering component 250 in connection with FIG. 2.

In some embodiments, one or more processors 830 may be packaged togetherwith DOI logic 822. In some embodiments, one or more processors 830 maybe packaged together with DOI logic 822 to form a System in Package(SiP). In some embodiments, one or more processors 830 can be integratedon the same die with DOI logic 822. In some embodiments, processors 830can be integrated on the same die with DOI logic 822 to form a System onChip (SoC).

Illustrative I/O components include a microphone, joystick, game pad,satellite dish, scanner, printer, display device, wireless device, acontroller (such as a stylus, a keyboard, and a mouse), a natural userinterface (NUI), and the like. In aspects, a pen digitizer (not shown)and accompanying input instrument (also not shown but which may include,by way of example only, a pen or a stylus) are provided in order todigitally capture freehand user input. The connection between the pendigitizer and processor(s) 830 may be direct or via a coupling utilizinga serial port, parallel port, Universal Serial Bus (USB) port, and/orother interface and/or system bus known in the art. Furthermore, thedigitizer input component may be a component separated from an outputcomponent such as a display device, or in some aspects, the usable inputarea of a digitizer may coexist with the display area of a displaydevice, be integrated with the display device, or may exist as aseparate device overlaying or otherwise appended to a display device.Any and all such variations, and any combination thereof, arecontemplated to be within the scope of aspects of the technologydescribed herein.

Computing device 800 may include networking interface 880. Thenetworking interface 880 includes a network interface controller (NIC)that transmits and receives data. The networking interface 880 may usewired technologies (e.g., coaxial cable, twisted pair, optical fiber,etc.) or wireless technologies (e.g., terrestrial microwave,communications satellites, cellular, radio and spread spectrumtechnologies, etc.). Particularly, the networking interface 880 mayinclude a wireless terminal adapted to receive communications and mediaover various wireless networks. Computing device 800 may communicate viawireless protocols, such as Code Division Multiple Access (CDMA), GlobalSystem for Mobiles (GSM), or Time Division Multiple Access (TDMA), aswell as others, to communicate with other devices via the networkinginterface 880. The radio communications may be a short-range connection,a long-range connection, or a combination of both a short-range and along-range wireless telecommunications connection. A short-rangeconnection may include a Wi-Fi® connection to a device (e.g., mobilehotspot) that provides access to a wireless communications network, suchas a wireless local area network (WLAN) connection using the 802.11protocol. A Bluetooth connection to another computing device is a secondexample of a short-range connection. A long-range connection may includea connection using one or more of CDMA, GPRS, GSM, TDMA, and 802.16protocols.

The subject matter of embodiments of the invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

For purposes of this disclosure, the word “including” has the same broadmeaning as the word “comprising,” and the word “accessing” comprises“receiving,” “referencing,” or “retrieving.” In addition, words such as“a” and “an,” unless otherwise indicated to the contrary, include theplural as well as the singular. Thus, for example, the constraint of “afeature” is satisfied where one or more features are present. Also, theterm “or” includes the conjunctive, the disjunctive, and both (a or bthus includes either a or b, as well as a and b).

For purposes of a detailed discussion above, embodiments of the presentinvention are described with reference to a head-mounted display unit;however, the head-mounted display unit depicted herein is merelyexemplary. Components can be configured for performing novel aspects ofembodiments, where “configured for” comprises programmed to performparticular tasks or implement particular abstract data types using code.Further, while embodiments of the present invention may generally referto the head-mounted display unit and the schematics described herein, itis understood that the techniques described may be extended to otherimplementation contexts.

Embodiments of the present invention have been described in relation toparticular embodiments which are intended in all respects to beillustrative rather than restrictive. Alternative embodiments willbecome apparent to those of ordinary skill in the art to which thepresent invention pertains without departing from its scope.

From the foregoing, it will be seen that this invention is one welladapted to attain all the ends and objects hereinabove set forthtogether with other advantages which are obvious and which are inherentto the structure.

It will be understood that certain features and sub-combinations are ofutility and may be employed without reference to other features orsub-combinations. This is contemplated by and is within the scope of theclaims.

What is claimed is:
 1. A wearable device, comprising: a computer memory and a processor in communication with the computer memory, the processor is configured to: cause display of a menu having at least three menu items corresponding to at least three different displacement ranges transversal to a longitudinal axis of the wearable device, the at least three menu items being configured in an arrangement based on respective angles of the at least three different displacement ranges; detect a movement of the wearable device from a first position to a second position with a displacement transversal to the longitudinal axis of the wearable device, wherein the arrangement of the at least three menu items on the menu remains unchanged during the movement of the wearable device; determine that the displacement corresponds to a displacement range of the at least three different displacement ranges; and activate, in a computer-mediated reality environment, an operation related to a menu item corresponding to the displacement range in response to the determination that the displacement corresponds to the displacement range.
 2. The device of claim 1, wherein the processor is further configured to determine contextual information related to the operation in the computer-mediated reality environment.
 3. The device of claim 1, wherein the processor is further configured to cause a visual feedback to present with the menu item in response to the displacement being corresponding to the displacement range.
 4. The device of claim 1, further comprising: a displacement sensor to measure a value of the displacement.
 5. The device of claim 4, wherein the processor is further configured to determine the value of the displacement being greater than a minimum distance and lesser than a maximum distance associated with the displacement range.
 6. The device of claim 1, further comprising: an orientation sensor to determine an angular direction of the displacement in reference to the first position being an origin of a two-dimensional Euclidean plane transversal to the longitudinal axis.
 7. The device of claim 6, wherein the processor is further configured to determine the angular direction being bound by a minimum angle and a maximum angle of the displacement range.
 8. The device of claim 1, wherein the processor is further configured to arrange the at least three menu items in a radial fashion according to the respective angles of the at least three different displacement ranges.
 9. The device of claim 1, wherein the processor is further configured to cause the menu being removed from a present view of the computer-mediated reality environment in response to the displacement being outside of all displacement ranges associated with the menu.
 10. The device of claim 1, wherein the processor is further configured to cause display a replica of the menu on a floor around a user of the wearable device.
 11. A computer-implemented method, comprising: causing display of a guiding menu having a plurality of menu items corresponding to different displacement ranges of a wearable device, the plurality of menu items being configured in a radial arrangement based on respective angles of the different displacement ranges; detecting the wearable device moved from a first position to a second position with a displacement transversal to a longitudinal axis of the wearable device, wherein the radial arrangement of the plurality of menu items on the guiding menu remains unchanged when the wearable device moved from the first position to the second position; determining the displacement being within a displacement range associated with a menu item of the plurality of menu items; and invoking an actionable item represented by the menu item in response to the displacement being within the displacement range associated with the menu item.
 12. The method of claim 11, further comprising: detecting another movement of the wearable device from the second position to the first position; and activating an operation associated with the actionable item in response to the another movement.
 13. The method of claim 11, further comprising: detecting a direction and a distance of the displacement; determining the direction being within a range of directions associated with the displacement range; and determining the distance being greater than a minimum threshold and less than a maximum threshold associated with the displacement range.
 14. The method of claim 11, further comprising: detecting another movement of the wearable device with a displacement parallel to the longitudinal axis, wherein the guiding menu is displayed in response to of the another movement being detected.
 15. The method of claim 11, further comprising: detecting a hand gesture; and interpreting the hand gesture based on the actionable item.
 16. The method of claim 15, further comprising: modifying an object or a view of a computer-mediated reality environment based on a combination of the actionable item and the hand gesture.
 17. The method of claim 11, further comprising: causing display of a replica of the guiding menu on a plane transversal to the longitudinal axis of the wearable device. 